Subject: CP Takeover requirement for APPN

From: "jim (j.) cobban" <[email-masked]>
Date: Mon, 4 Jul 1994 13:03:00 -0400

The current APPN architecture model assumes that all processes are running in
a single processor.  An externally induced failure which takes down one
process will therefore, naturally, take down all processes.  However
increasingly achieving performance and availability goals in an economical
manner leads to the implementation of multiple-processor systems.  In some
cases these systems may have a common memory system which permits dynamically
switching processes to a still running processor, but this increases costs
substantially, and may introduce yet another single point of failure.  We
therefore need to consider the impact on the APPN process model of operating
in an environment where the processes are distributed across multiple
processors which do not share memory space, and may not even be physically
co-located, as long as they can exchange the signals specified in the APPN
architecture.

Communications Intensive NN
---------------------------
Processes can be assigned to separate processors in many ways as long as they
do not share memory.  The particular functional split which I wish to address
here is one appropriate to a communications intensive implementation,
particularly a network node which is dedicated to ISR traffic.

In this application the bulk of the CP (DS, TRS, CS, SCM, ASM) runs on one
processor while the ISR data path (DLC, PC, SC, AS) runs on a different
processor.  Note that the ASM/AS split which is being separately considered
is essential for this application.  There is also a requirement for a split
either part way through CS or part way through DLC.  A split in CS provides a
better balance of functionality, with the ISR data path having more
independence of action on XID exchanges.  There are others with more
experience in that area who can provide a better analysis.  I will discuss
a split in DLC so that the existing CS-DLC interface is retained. 

ISR processor failure scenario
------------------------------
If the processor which is running the ISR data path fails then the following
notifications are required:

 - CS is notified by DLC that the port has failed by an ALERT_MSU
 - CS is notified that the PC process has died.  CS sends DEACTIVATE_AS to
   ASM.
 - ASM sends SESSION_ROUTE_INOP to SCM as a result of the DEACTIVATE_AS
   from CS (no change).
 - SCM destroys all of the instances of SC which used the failed AS/PC 
   (no change).  Most of these are probably already destroyed.  SCM sends
   UNBIND in the opposite direction (if possible).

When this processor comes up, or is replaced, reactivation is normal.
The only change to the architecture is that CS receives a new signal from
"PC" indicating that PC has spontaneously died. 

CP processor failure scenario
-----------------------------
If the processor which is running the CP fails while the processor which is
running the data path is still operational, the data path can continue
running.  New sessions cannot be established because DS and TRS are
unavailable, but there is no obvious reason to take down the existing
sessions.  When a new CP becomes available, in the currently architecture of
record there is no mechanism to resynchronize knowledge of the state of the
PC, AS, and SC processes, and therefore we are forced to take them down.

This is counter-intuitive for our customers and conflicts with the
implementation of the composite network node in VTAM V4 and NCP V6.

Some of the things that are required are:

 - When the new CS attempts to ACTIVATE_PORT for a port which is already
   running it should receive back information on already existing
   connections, including the process identification of the PC process
   associated with each connection.  This is information similar to what is
   currently signaled on CONNECT_IN (RQ) from the DLC and SET_MODE(RQ) from
   CS.
 - On receipt of a SET_MODE(RQ) from DLC CS asks PC to ask AS to contact ASM.
   CS knows the process id of the new instance of ASM but does not know the
   process id of the old instance of AS.
 - ASM provides AS with the new process id of SCM.  
 - AS provides SCM with the process ids of all the instances of SC.
 - SCM communicates with the instances of SC to recover the session
   information.

Optimization
------------
The volume of message traffic is significant.  However this is a rare
occurrence and even if it takes several minutes to resynchronize the
processes that is less disruptive than tearing down existing sessions.

SCM could be split such that the table of active SCs is distributed.
Reestablishing session awareness would then only require the global SCM
acquiring knowledge of the process id of the processor specific SCM.

If CS is split so that port specific processing is performed on the same
processor that the port is on, then the amount of traffic inside CS would be
simplified: only new process ids would need to be exchanged.  The big problem
with CS is that the algorithm for assigning TG numbers is not capable of
being distributed.

---

From: "jim (j.) cobban" <[email-masked]>
Date: Tue, 26 Jul 1994 14:32:00 -0400

My original message was deliberately sketchy about details, because it was my
understanding that the SIG wanted a one page brief overview of the nature of
the problem, not a completely fleshed out solution.

>ISR processor failure scenario

This flow is shown in Figure S-55 on page S-99.  The only implementation
action which is required is in the DLC, and therefore outside of the
architecture.  DLC generates an ALERT_MSU to signal to CS the failure of each
link station which was connected through the processor which has failed. 
Receipt of this ALERT_MSU is sufficient to drive the entire cleanup.  The
second bullet in my original note is wrong.  CS sends the DEACTIVATE_AS as a
result of receiving the ALERT_MSU.  The only purpose of notifying CS that the
PC child process is no longer there is to skip the destroy signal to the
child.  This concern could equally be handled in this case by the underlying
process scheduler simply discarding destroy signals for processes which it
knows are already dead or unreachable.  This would also avoid modifying SCM
to understand that it doesn't have to send destroys for the missing instances
of SC.

>CP processor failure scenario

+----+ +----------------------------------------------+ +---------------------+
|    | |   Main Processor                             | | ISR Processor       |
|Node| |+---+ +--------------------------------+ +---+| |+----+ +--+ +--+ +--+|
|Op- | ||NOF| |            Control Point (CP)  | |DLC|| ||Port| |PC| |AS| |SC||
|er- | ||   | |      TRS       CS  ASM  SCM    | |   || ||    | |  | |  | |  ||
|ator| |+---+ +--------------------------------+ +---+| |+----+ +--+ +--+ +--+|
|    | |                                              | |                     |
+----+ +----------------------------------------------+ +---------------------+
   .      .           .        .    .    .         .       .     .    .    .
   .START_PORT(RQ)    .        .    .    .         .       .     .    .    .
   o----->o           .        .    .    .         .       .     .    .    .
   .      |START_PORT(RQ)      .ACTIVATE_PORT(RQ)  .       .     .    .    .    
   .      +------------------->o------------------>o------>o     .    .    .
1  .      .           .        .    . "Old Connection (RQ)".<-+  .    .    .
   .      .           .        o<------------------o<------o  |  .    .    .
   .      .           .        ."Old Connection (RSP)"     .  |  .    .    .
   .      .           .        o------------------>o------>o  |  .    .    .
   .      .           .        .    .    .         .       +--+  .    .    .
   .      .           .        .    .    ACTIVATE_PORT(RSP).     .    .    .
   .      .           .        o<------------------o<------o     .    .    .
		   See Figure S-13 on page S-21
   .      .           .        .    .    .         .       .     .    .    .
2  .      .           .        ."Contact ASM"      .       .     .    .    .
   .      .           .        o-------------------------------->o    .    .
   .      .           .        .    .    .         .       .     |    .    .
   .      .           .        .    .    .         .       .     +--->o    .
3  .      .           .        .    .    .         .  "AS Active (RQ)"|    .
   .      .           .        .    o<--------------------------------+    .
   .      .           .        .    |"AS Active (RSP)"     .     .    .    .
4  .      .           .        .    +-------------------------------->o    .
   .      .           .        .    .    .         .       .      "New|SCM".
5  .      .           .        .    .    .         .       .     .    +--->o
   .      .           .        .    .    .         .       .    "SC Active"|
6  .      .           .        .    .    o<--------------------------------o
   .      .           .        .    .    .         .       .     .    .    .

  Annotations:

  1   A new message is sent back to CS by DLC to report the existence of
      a previously established link station connection, complete with PC
      process identification.  One of these messages is sent for each
      existing connection.  These messages are sent prior to the
      ACTIVATE_PORT response to prevent CS from attempting to CONNECT_OUT to
      stations.  "Old Connection" is only sent by DLC for link stations
      which are already connected, and therefore have PC and AS processes.
      For other stations which were not already connected at the time the
      main processor failed normal CONNECT_IN or CONNECT_OUT processing is
      followed.

  2   For each "Old Connection (RQ)" received CS asks PC to ask AS to
      contact ASM.  CS is out of the circuit now, so no response is required
      from PC to CS.

  3   AS contacts ASM to provide it with its existing process ID.  

  4   ASM provides AS with the new process id of SCM.

  5   AS provides each instance of SC with the process id of the new instance
      of SCM.

  6   Each SC identifies itself to SCM, and provides the session information.

Note the minor change from my first proposal at steps 5 and 6.

Not dealt with is how the isolated ISR handles the inability to communicate
with the CP during the interruption.  PC, SC, and "Port" have no protocol
boundaries with the CP which would be active during this interruption except
for PC sending alerts to MS and "Port" sending alerts to DLC.  AS rejects
BINDs received from the link if its protocol boundaries with the CP are
unavailable.